Optimizing Data Movement in Hybrid Analytic Systems

i Dedication ii Acknowledgements iii List of Tables vi List of Figures vii CHAPTER 1: INTRODUCTION 1 CHAPTER 2: ASSUMPTIONS, DEFINITIONS, AND BACKGROUND 16 2.1. Assumptions 16 2.1.1. List of Foundational Observations and Assumptions 16 2.1.2. Discussion 22 2.2. Definitions 23 2.2.1. Agrios-Specific Terms and Concepts 23 2.2.2. Terms and Concepts in Relational Query Processing 29 2.3. Background 36 2.3.1. R, Array Databases, and SciDB 36 2.3.2. Hybrid Analytic Systems 39 2.3.3. Query Optimization 46 2.4. Conclusion 52 CHAPTER 3: AGRIOS’ CONCEPTUAL MODEL 53 3.1. Agrios as Optimizer 56 3.2. Staging 59 3.2.1. Cost Model 59 3.2.2. Transformation Types 61 3.2.3. Search Engine 67 3.3. Discussion 92 3.3.1. Search-Space Representation 92 3.3.2. Particular Refinements 95 3.3.3. Why Bonneville? 96 3.4. Conclusion 97 v CHAPTER 4: AGRIOS IMPLEMENTATION 98 4.1. Components of Agrios 98 4.1.1. R 98 4.1.2. SciDB 101 4.1.3. Motivation 103 4.2. Agrios as Integration 105 4.2.1. Scope 105 4.2.2. Architecture 118 4.3. Configuration Options 132 4.4. Conclusion 133 CHAPTER 5: THE CASE FOR STAGING 134 5.1. General Overview 134 5.2. Staging Costs 134 5.3. Examination of Plan Costs 142 5.3.1. Overview 142 5.3.2. Methodology 145 5.3.3. Results 149 5.3.4. Additional Considerations 158 5.4. Conclusion 171 CHAPTER 6: EXPERIMENTAL EVALUATION 173 6.1. Overview 173 6.2. Methodology 174 6.3. Results 174 6.3.1. Staging 174 6.3.2. Transformation and Query Rewriting 177 6.3.3. Query Accumulation 181 6.4. Rule Types and Staging 184 6.5. Conclusion 197 CHAPTER 7: CONCLUSION AND FUTURE WORK 199 7.1. Conclusion 199 7.2. Future Work 201 7.2.1. Extensions to Agrios 201 7.2.2. Applications to Other Settings 210 References 213